Memorization and generalization in neural code intelligence models

Deep Neural Networks (DNNs) are increasingly being used in software engineering and code intelligence tasks. These powerful tools that capable of learning highly generalizable patterns from large datasets through millions parameters. At the same time, their capacity can render them prone to memorizing data points. Recent work suggests memorization risk manifests especially strongly when training dataset is noisy, involving many ambiguous or questionable samples, only recourse. The goal this paper evaluate compare extent generalization neural models. It aims provide insights on how may impact behavior models systems. To observe models, we add random noise original use various metrics quantify aspects testing. We several state-of-the-art benchmarks based Java, Python, Ruby codebases. Our results highlight important risks: trainable parameters allow networks memorize anything, including noisy data, a false sense generalization. observed all manifest some forms memorization. This be potentially troublesome most tasks where they rely rather noise-prone repetitive sources, such as GitHub. best our knowledge, first study effects domain raises awareness provides new into issues systems usually overlooked by researchers.